Huffman Codes Lecturer : Michel

Shannon’s noiseless coding theorem tells us how compactly we can compress messages in which all letters are drawn independently from an alphabet A and we are given the probability pa of each letter a ∈ A appearing in the message. Shannon’s theorem says that, for random messages with n letters, the expected number of bits we need to transmit is at least nH(p) = −n ∑ a∈A pa log2 pa bits, and there exist codes which transmit an expected number of bits of just o(n) beyond this lower bound of nH(p) (recall that o(n) means that this function satisfies that o(n) n tends to 0 as n tends to infinity). We will see now a very efficient way to construct a code that almost achieves this Shannon lower bound. The idea is very simple. To every letter in A, we assign a string of bits. For example, we may assign 01001 to a, 100 to d and so on. ’dad’ would then be encoded as 10001001100. But to make sure that it is easy to decode a message, we make sure this gives a prefix code. In a prefix code, for any two letters x in y of our alphabet the string corresponding to x cannot be a prefix of the string corresponding to y and vice versa. For example, we would not be allowed to assign 1001 to c and 10010 to s. There is a very convenient way to describe a prefix code as a binary tree. The leaves of the tree contain the letters of our alphabet, and we can read off the string corresponding to each letter by looking at the path from the root to the leaf corresponding to this letter. Every time we go to the left child, we have 0 as the next bit, and every time we go to the right child, we get a 1. For example, the following tree for the alphabet A = {a, b, c, d, e, f}: